Local Intrinsic Dimensional Entropy

Most entropy measures depend on the spread of probability distribution over sample space |X|, and maximum achievable scales proportionately with cardinality |X|. For a finite this yields robust which satisfy many important properties, such as invariance to bijections, while same is not true for continuous spaces (where |X|=infinity). Furthermore, since R R^d (d in Z+) have (from Cantor's correspondence argument), cardinality-dependent cannot encode data dimensionality. In work, we question role defining spaces, can undergo multiple rounds transformations distortions, e.g., neural networks. We find that average value local intrinsic dimension distribution, denoted ID-Entropy, serve measure capturing ID-Entropy satisfies desirable properties be extended conditional entropy, joint mutual-information variants. also new information bottleneck principles links causality. context deep learning, feedforward architectures, show, theoretically empirically, hidden layer directly controls generalization gap both classifiers auto-encoders, when target function Lipschitz continuous. Our work primarily shows that, taking structural rather than statistical approach preserve dimensionality, being relevant studying various architectures.